-
-
Notifications
You must be signed in to change notification settings - Fork 31.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixed #373 -- Added CompositePrimaryKey. #18056
base: main
Are you sure you want to change the base?
Conversation
I was trying out this exciting branch and ran into this error when running a test:
The issue stems from the use of
Curious if anyone ran into this as well. Edited for traceback:
So, this is part of |
Thanks for testing and reporting the issue @grjones! Indeed, I forgot to handle this use case. I'll look into it this week. |
c75dcdd
to
4be1c68
Compare
@grjones, FYI I pushed the fix |
Nice! I hope this gets merged in soon. Your branch has been working great for me. |
I may have found one other small issue. When adding a regular |
@grjones , thanks, I appreciate the feedback, I'll look into it. If a model defines |
I'll see if I can give you a solid failing test. My "unique constraint" phrasing might not be exactly right. But ultimately, I believe Django queries the DB first to see if the new object's PK already exists and throws a validation error. The composite key logic doesn't seem to be doing that and so an unhandled IntegrityError is raised instead. |
@grjones , sorry for the late reply, I've been busy last week. Could you give me more specifics? What's the error message you expect? |
Actually, I think it's mostly ok. I was using Django Spanner and it's just not quite working with composite keys and will need to be fixed there. I wrote this and it passed. It probably shouldn't say
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a great start!
I've left a bunch of ideas for improvement. Feel free to push back if you think I'm wrong about anything.
Thank you so much for taking the time to review my changes @LilyFoote !
|
I don't feel strongly that this is better or worse than another option here, so happy to go with what you think is best.
I like your tests quite a bit - they're pretty readable and comprehensive. The main issue I have with them is that they're written for specific databases instead of for generic database features. Where possible Django strongly prefers to test based on features because then the tests apply to as many databases as possible (including third party database libraries). I think the asserts of the actual SQL might be a bit tricky to adapt though, so we might need a different way to check what they're checking. Also, after I reviewed yesterday, I thought of some more things:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking better and better. I have a few new comments and questions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a few comments here and there. It's great to see the progress you've made on combining MultiColSource
🎉
Something that came through my mind while reviewing is that we likely want a plan to eventually deprecate Options.pk
in favor of Options.primary_key
?
Thanks @charettes !
I'm not sure what you mean by that, I don't think we can, because |
So as far as I understand, at the moment I'm not sure it's the right decision to reuse this for composite fields, which on the other hand don't need Let me know what you think! |
You're completely right. In this case is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have two minor points about error message wording. They're not urgent to address.
It would not be set, if it's a regular primary key, |
I'll resolve the conflicts and rebase this after #18674 is merged. |
9c47271
to
7e11755
Compare
Hey everyone! We have managed to tackle some issues in separate PRs. Any review is appreciated. |
@charettes , could you take a look at this when you have the time? We'll need your approval before proceeding (as per Sarah's request). Any feedback is appreciated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great work @csirmazbendeguz and nearing the finish line 🏁
I reviewed everything except the tests themselves as I remember having another go at it. Let me know if you'd like me to review something in particular that I might have missed.
def _pk_constraint_sql(self, columns): | ||
return self.sql_pk_constraint % { | ||
"columns": ", ".join(self.quote_name(column) for column in columns) | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that might have been an opportunity to always use the PRIMARY KEY
constraint syntax even for single column primary keys (instead of the PRIMARY KEY
next to the column) but that should do for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could do a refactoring later, I didn't want to deal with it in this PR. I'm not sure if it's worth changing
django/db/models/aggregates.py
Outdated
assert fields == pk_fields, ( | ||
"Count doesn't support counting composite fields that are not composite" | ||
"primary keys" | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should raise a NotSupportedError
instead of an AssertionError
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, as far as I understand, based on the discussion #18485 (comment) , we shouldn't do these type of checks in Django. Since the only way to reach this path is via composite primary keys, there's no need to check.
django/db/models/base.py
Outdated
and field.column is None | ||
or field.generated | ||
or isinstance(field, CompositePrimaryKey) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't this already covered by field.column is None
? It'd be great to avoid an instance check here as this is on the hot path for model instance creation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue is there's an and
.
field.attname not in kwargs
and field.column is None
Maybe we could remove this line?
field.attname not in kwargs
I pushed a commit to remove the line. It doesn't break any tests.
Thanks @charettes ! I have addressed everything, please check my replies 👍 |
) | ||
|
||
cols = expr.get_cols() | ||
return Count(cols[0], filter=result.filter) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a small note that if we add support for composite fields in the future and they rely on ColPairs
as well this code will break as there might NULL
in there.
This currently only works because all cols of a CompositePrimaryKey
which has references to it turned into ColPairs
must be NOT NULL
so counting one member of the composite is equivalent to counting all of them (remember the SQL COUNT
function returns the number of non-NULL values)
To support non-primary key composite fields this would require doing
cols = expr.get_cols()
cols_filter = result.filter
for col in cols[1:]:
cols_filter &= Q(IsNull(col, False))
return Count(cols[0], filter=composite_filter)
I guess we could also only do it if columns are potentially NULL
today which would keep the optimization around when unnecessary
cols = expr.get_cols()
cols_filter = result.filter
for col in cols[1:]:
if col.output_field.null:
cols_filter &= Q(IsNull(col, False))
return Count(cols[0], filter=composite_filter)
But there would be no way to test it until non-primary key composite field support lands.
Last interesting note on the subject, recent versions of Postgres elide IS NULL
and IS NOT NULL
when possible
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I think for composite PKs we should count one field, and for other composite fields in the future do it the way you're proposing. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left a comment about the Count
implementation that is dependent on ColPairs
being from an origin when all cols are not col.output_field.null
(which is always the case for CompositePrimaryKey
). I'm really glad we added that as it will make integration with Pagination
and the admin (which relies on count
) much easier.
I'll give it a statuary approval given @sarahboyce and you have the documentation under control 🔍 🧙✅.
Congrats on getting this project so far @csirmazbendeguz and looking forward to getting it merged and the follow up tweaks and adjustments that will come from it 🎉 🏅
Amazing, thank you @charettes ! 🎉 |
74e5a19
to
d13c5e0
Compare
buildbot, test on oracle. |
d13c5e0
to
d1d50f7
Compare
I pushed a few very small docs tweaks, the diff can be seen here, if anyone doesn't like those feel free to add some comments |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @csirmazbendeguz 🎉
I will wait a day or two in case anyone else wants to jump in with feedback, I plan to land this on Monday 👍
Thank you @sarahboyce ! The docs look good to me 👍 |
I see nothing about whether or not this field is supported in forms. |
@timgraham, you're right. It supports the model's |
Just an explanation in the documentation. What happens if you try to use such a model in forms, formsets, etc. And be more explicit about what lack of admin support means... that a model that uses the field can't be registered. |
I added a section to warn users on correct usage with forms 👍 |
docs/howto/composite-primary-key.txt
Outdated
To avoid issues, make sure :attr:`~django.db.models.Field.editable` | ||
is set to ``False`` on all primary key fields. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can probably raise an error if this isn't the case - maybe using the checks framework.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah we could add a check to force composite PK fields to be editable=False
.
I'm just not sure if we should. This is an issue with regular primary keys as well: https://code.djangoproject.com/ticket/2259 . So if we force this for composite PKs , shouldn't we force it for regular PKs too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a skim through that issue and it seems the sticking point is creation vs updates - it should be possible to set the pk values during create and not after that. I'm not sure if Django has anything to handle that cleanly. If we don't, I'm happy to say this is out of scope (and in scope of 2259 instead).
1e434e4
to
ebc3bf6
Compare
Thanks Lily Foote and Simon Charette for reviews and mentoring this Google Summer of Code 2024 project.
ebc3bf6
to
46c77ed
Compare
@timgraham @smithdc1 I believe your comments have been addressed, thank you for the reviews! |
Trac ticket number
ticket-373
Branch description
This branch adds the
CompositePrimaryKey
field. If present, Django will create a composite primary key.Please refer to the docs for a more in-depth explanation.
Proposal
Previous PR
Composite FK
Admin
Composite Generic FK
Model._is_pk_set()✅Tuple Lookups✅Serial Fields❌Checklist
main
branch.